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ABSTRACT 

Arguing that relevcuice judgments are fundamental to 
the design and evaluation of all information retrieval systems, and 
that a consensus on the definition of the central concept of 
relevance has not been reached, this paper begins by criticctlly 
reviewing four approaches to the problem of defining relevance: (1) 
the system-oriented approach (relevance is defined as topicality, or 
matching) ; (2) the user oriented approach (relevcuice is defined as 
the relationship of information in a document to the research at 
hand, or in terms of user satisfaction) ; (3) the multidimensional 
approach (relevance is defined as topicality, as a concept contained 
within utility) ; and (4) the cognitive approach (which depends on 
concepts such as knowledge state, conceptual framework, and internal 
representation) . It then introduces the user-concentric approach, 
which is based on three assumptions: relevcuice is a multidimensional 
concept based on the human judgment process; it is dependent on both 
internal (cognitive) and external (situational) factors; and it is 
inter subjective but nevertheless systematic and measurable. Currently 
being explored at Syracuse University, this new approach is described 
as a model which explores relevance in the context of the 
sense-making approach, and places the user, not the system, in the 
role of the central and active determinant of the dimensions of 
relevance. (25 references) (CGD) 
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TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)," 



flBSTRflCT 



Althouph relevance judgaents are fundanental to the design and 
•valuation of all information retrieval systems, information scientists have 
not reached a consensus in defining the central concept of relevance. This 
paper re-examines the viewpoints of major writers in the field, then 
introduces a new approach. He believe that relevance is a multidimensional 
concept; that it is based on the human judgment process; that it is 
dependent on both internal (cognitive) and external (situational) factors; 
and theit it is intersubjcctive but nevertheless systematic and measurable. 
Ue suggest a different paradigm from the classic source-tc^^-destination 
communication model of information retrieval. This user-centric cognitive 
model views the user — not the system — as the central and active 
determinant of the dimensions of relevance. 
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IWIRODUCTION 

&iTM» informition science first began to coalesce into ^. distinct 
discipline in the forties and early fifties, ralevance has been identified 
as its fundaaental and central concept. Yet serious questions about the 
definition and nature of relevance remain. 

^Intuitively, Me understand quite well what relevance means,'* says 
Saracevic. "It is a primitive 'y'knoN' concept, as is information for which 
Me hardly need a definition. ...if and Mhen any productive contact Cin 
communication] is desired, consciously or not. Me involve and u^e this 
intuitive notion of relevance." til 

Despite, or perhaps because of, its intuitive nature, relevance has 
been used explicitly and implicitly in ev&iuation of information systems. In 
fact, a huge body of evaluation literature has accumulated that uses 
relevance without thoroughly understanding Mhat it is. Another body of 
literature attempts, unsuccessfully, to define the concept. In the late 
fifties information scientists engaged in intense debates on theoretical and 
methodological aspects of relevance.. In the sixties tMO major studies 
examined the factors that influence relevance. This research was summarized 
in the mid--sevent ies, and since then, overt interest has seemed to subside. 
Yet the need for a definition is more pressing than ever, for at least three 
reasons: 

(1) Relevance is the basis of all information retrieval systems, 
including full-text systems, question-answering systemsi database management 
systems, and knowledge-based systems. Increasingly complex information 
retrieval systems are being developed that promise to serve users faster, 
more efficiently, and more directly than ever. It is inevitable that these 
new systems will be evaluated, as have systems since the library card 
catalog, on the basis of human relevance judgments. Without a firmly 
established understanding of relevance, it seems difficult (if not 
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impossible) to make definitive statements about systems or system features 
or to make comparisons across systems. 

(2) ftwong current developments are information retrieval systems that 
actually respond to the user's relevance judgments. Relevance feedback, for 
example, makes the user an integral part of the system. In such a system, 
relevance is no longer a reactive concept, to be used primarily in 
evaluation, but an active concept vital to the functioning of the system 
itself. But what is this "relevance" that is being actively used? That is, 
it seems necessary to understand what users "mean" by relevance if a 
feedback mechanism is to make best use of user involvement, 

(3) Information scientists must finally establish a theoretical 
understanding of relevance as a fundamental concept, so that the discipline 
of information science can move on to other matters. 

Two key questions are at issue here: What is relevance? And, what is 
the role of the user in judging relevance? In an attempt to revive interest 
in the problem of defining relevance, this paper critically reviews 
tipproaches to these questions and the assumptions on which they are based, 
then describes a recent approach that may offer a solution. 

SYSTEM-ORIENTED flPPRQflCHES 

Information science is traditionally based on the classic model that 
describes communication as the transfer of a message from a source to a 
destination — in information retrieval terms, from a system to a user. The 
practical goal of information science has been to design systems to retrieve 
information that satisfies the request of the user, that is, information 
that is relevant to the user. The clearest and most persistent definition of 
relevance is one of matching or topicality : whether the topic of the 
information retrieve?! matches the topic of the request. 

Fundamentally, information retrieval systems have relied on the 
matching of words. Further enhancements have focused on counting or 



measuring the frequency with which terns that describe the content of a 
document occur, or how relevant t?r«s tend to cluster in certain linguistic 
patterns or sets. Considerations such as these are also important for 
systems design in natural language processing, automatic indexing and 
abstracting, and full-text retrieval. Many models are probabilistic, for 
example, those that assign numprical values or weights to the terms in 
documents according to their probability of being judged relevant by the 
user. Applications include systems that rank documents for the user in order 
of their relevance, and relevance feedback systems in which term weighting 
may be used to modify lists of documents retrieved as the search progresses. 
Still other models derive from formal logic, such as those concerned with 
developing decision rules for indexing. 

Overall, these models, regardless of their complexity and 
sophistication, share a fundamental assumption of matching. Topical 
relevance lies at the heart of the design and evaluation of information 
retrieval systems. Topicality is both intuitively valid and practical: 
applicable, observable, and measurable. 

But topicality is insufficient as a definition of relevance because, 
while it depends on matches between queries and documents (or document 
terms), it does not necessarily encompass the information needs of the user. 
Furthermore, a topical definition does not represent the full content and 
context of documents. Certainly the importance of the user in making the 
ultimate relevance decision has been recognized as far back as 1956 at the 
International Conference for Scientific Informati on in Washington, D. C. In 
separate papers, Vickery C2, 33 presents views on two kinds of relevance: 
relevance to a sub.iect and user relevance. Relevance to a subject most 
closely fits the topicality definition, or matching queries with documents. 
In user relevance, the user decides how far to pursue the search for 
inf ormat ion. 
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USER-QRIENTED QPPRQftCHES 

After topicality, the term usefulness probably appears most often in 
the relevance literature. Usefulness generally refers to some Nay in which 
information retrieved fulfills a user's need. Saracevic LAI includes in his 
summary list of 16 definitions the notions that relevancre is "a measure of 
usefulness of an answer" and "an indication of significance to an important 
purpose." Relevant materials or information include ''ideas or facts so 
closely related to the problem at hand that disregarding them would alter 
the problem"; "ideas or facts useful in considering the matter at hand"; and 
"answers of use in current work. " 

Rees and Schultz C53, in their large-scale experimental study of the 
relevance judgment process, place relevance and usefulness in separate but 
overlapping classes. According to their definition, relevance is the 
relationship of information in a document to the research at hand. Relevance 
is contained within usefulness , which also relates to users* existing 
knowledge and individual characteristics. After post-experimental 
questioning of their subjects, however, Rees and Schultz conclude, "It is 
clear that the definitions of relevance and usefulness were not mutually 
exclusive, and to the degree that the judges were sensitive to this overlap, 
they may have experienced some difficulty in discriminating between the two 
terms. " 

Another approach defines relevance in terms of user satisfaction. 
Generally, satisfaction relates to the user's state of mind or personal 
attitude as expressed in response to somi^ aspect or aspects of an 
information search. It is often operational iznd as a measure in performance 
evaluations of information retrieval systems. Satisfaction measures tend to 
be compiled from questionnaires that ask users to rate their degree of 
satisfaction with widely varied aspects of search, retrieval system, and 
system environment. Thus satisfaction may be a composite that contains 




judgments of relevance. Both theorists and system evaluators seem to agree 
that satisfaction is an extremely complex concept, one that relates system 
output and attributes of the system itself to the user's information need. 
C6, 7] 

Many more terms (e.g., appropriateness, fit, aboutness, pertinence) 
have been used in attempts to establish a user-oriented definition of 
relevance. E8, 9, 10, 11, 12]. But taken together, these attempts appear to 
confuse rather than clarify the meaning of the concept. A more general 
approach is to define relevance as a relationship, as Saracevic C4] does in 
the form of an algorithm: "Relevance is the (fl) gage of relevance of an (B) 
aspect of relevance existing between an (C) ob.iect .ludoed and a (D) frame of 
reference as judged by an (E) assessor. " With appropriate terms inserted 
from the lists he provides, the statement might read: "Relevance is a 
measure of utility existing between a document and a question as judged by a 
requester , " This algorithm, Saracevic says, '*demonstrateCs] almost every 
definition ever used for the concept of relevance." 

In short, no single view has emerged as a "final" definition of 
relevance. But of the definitions described so far, all except topicality 
depend on the user — not the system — to determine what information is 
relevant. And all imply a definition of relevance as some sort of 
relationship or relationships. 

MULTIDIMENSIONAL flPPRQflCHES 

The apparent complexity of the concept of relevance seems to point to a 
multidimensional definition. For example. Cooper C13] defines relevance as 
topicality, a concept conta.'ined within utility. Utility, he says, is a 
"catch-all concept" ... a "cover term for whatever the user finds to be of 
value about the system output, whether its usefulness, its entertainment or 
esthetic value, or anything else." Cooper does not isolate or test these 
factors systematically, but in several papers C13, 14, 15, 163 mentions 



qualities such as novelty; informativeness; credibility (publication source, 
authorship, recency); importance or weighting of components in request; 
clarity (ease with which relevance can be detected by user or systea) ; 
involvement with the system (time spent, effort expended); possible negative 
factors (boredom, unpleasantness occasioned by thd content of the docuinent); 
and possible esthetic factors (witty style, beauty of illustrations). Based 
on all these factors, Coopcr^s definition oF utility night also be viewed as 
a multidimensional concept of relevance. 

In the sixties, two major studies actually based on a multidimensional 
view identified sets of potential relevance variables and tested their 
effects and relationships. The studies of Rbes and Schultz C5] and Cuadra 
and Katter C173, which each identified about 40 variables, yielded more 
collective insight into the factors affecting relevance than any one study 
since, but they failed to resolve the question of definition. They did, 
however, find the judge to play an important role. 

Rees and Schultz C5] define relevance as "a relation between system 
responses and user request established by a judgment made by the user or his 
delegate." The results 01 their study show that individual differences most 
strongly affect relevance judgments. In their conclusion, Rees and Schultz 
say, "Although the initial search should be for variables related to the 
relevance assessm^^nts, the most fruitful findings will appear when 
statements are possible about personal characteristics which lead to, or 
cause, certain types of relevance ratings." 

Cuadra and Katter C17] see the ** judgmental phenomenon" of relevance as 
indicating, first, the relationship of a term or document to the user's 
fi^ld of interest (implicit use orientation); and second, the relationship 
between system output, such as a document, and the user's information 
requirement, either specified or assumed. The results of their study show 
the judge to play a central role in determining relevance. They conclude by 



emphasizing the importance of understanding users' "real needs" and 
suggesting that a major priority for future research lies in developing 
models of user and situation. 

The seventies were marked by the impressive efforts of Saracevic CI, 43 
to establish a definitional framework for relevance as a theoretical 
construct through summarizing, refining, and developing earlier views. The 
notion of relevance, Saracevic C43 says, is directly connected to the model 
of information retrieval as a source^to-destxnat ion communication process. 
Relevance "is a measure of effective contact between a source and a 
destination." Using Cuadra and Katter's [173 framework, Saracevic groups all 
experimental variables from the literature into five major classes: 

(1) documents and document representations} 

(2) queries; 

(3) judgmental situations and conditions; 

(4) mode of expression (rating scales); and 

(5) judges (people). 

The most important category, Saracevic C43 says, ;;s people: •'It is clear 
that the other four classes of variables ... relate more or less to people 
in at least two respects: either how people (judges, users, user delegates, 
system operators, etc.) were affected by them in their judgment or how these 
variables fared from some action by people (e.g., relation between texts and 
answers, distribution of judgments over documents and document 
representations). " 

When Saracevic synthesizes the results of the experimental studies, he 
concludes thai; subject knowledge is the most important variable concerning 
people: the less people know, the more items they judge relevant. Within 
groups of subjects, situational factors (e.g., professional involvement with 
the problem, intended uses of documents, the system being being 
evaluated/used) generally affect agreement in relevance judgments. 

Unfortunately, major criticisms of relevance research cast doubt on 
even these findings. Saracevic 1:43 himself comiisnts that experiments on 
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relevance have been conducted without significant connection to or from 
theories. Certainly control in experimental situations presents a difficult 
challenge. In evaluating the performance of retrieval systems and in 
studying user behavior, researchers typically use artificial test 
collections and simulated search conditions. They predetermine Nhat ar£ 
"relevant" documents or "correct" relevance judgments as standards for 
measurement. The assumption is that meaning is inherent in text and that 
experts are able to extract it and apply it to retrieval problems. Users, 
then, essentially are forced to translaiO their information needs into the 
standard categoi^ies of the existing information system. The results of these 
experiments do not necessarily shed light on the behavior of real users 
searching real collections in real situations. 

COGNITIVE flPPROflCHES 

ft number of promising perspectives that apply to questions of relevance 
come from the psychological disciplines. Information scientists who have 
adopted and developed the approaches of cognitive psychology tend to view 
relevance judgments as intersubject ive and constantly evolving phenomena. 
They see users as taking responsibility for — and an active role in — 
making judgments. Users' judgments, in turn, are affected by any or all 
internal factors (e.g., attitudes and prejudices) and external factors 
(needs and situations) that influence them as individuals. Cognitive 
approaches, which depend on concepts such as knowledge state^ conceptual 
framework, and internal representation, clearly represent a trend away from 
system's views and toward user's views of relevance. 

Saracevic C43 suggests these trends when he proposes three hypotheses: 

(1) "Only the user himself may judge the relevance of the documents to 
him and his uses"; 

(S) "For the same user a relevance judgment may change over time"; 

(3) "Various types of judgments may exist because of the different 
purposes for which information is required." 
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These hypotheses not only center on the user, but also go beyond the user. 
They iiiply that relevance judgment depends entirely on the perception of the 
user and the situation of the moment. So far, these hypotheses have not been 
thoroughly tested. 

On the other hand, several interesting theories have evolved that 
relate not to the definition of relevance per se, but to information needs 
and uses. These theories tend to look less at what is specified in 
information queries than at what is not. 

For instance, flrtandi Ci63 views information as a means of reducing 
uncertainty: "To reduce uncertainty information must be * relevant' in the 
sense that it can be integrated and evaluated by the individual in terms of 
prior experiences (his existing state) and his possible future states and 
activities." wn approach called anomalous states of knowledge (ASK), is 
described by Belkin, Oddy and Brooks C193. ASK sees the user as being in an 
anomalous state of knowledge in which he actually may be unable to 
articulate what is needed to solve his problem. Taylor C203 describes the 
user-values approach^ which focuses on the user's problem or problem 
situation; on perceptions of utility and value that the user brings to the 
system. Finally, the interdisciplinary sense-making aoproachi developed 
primarily by Dervin C213 concentrates on how people bridge cognitive gaps in 
order to make sense of their worlds. This approach allowu an active role for 
the judge in a cognitive, non-physical environment in which conditions 
constantly change. 

THE USER-CENTRIC APPROACH 

Eisenberg, Nilan, and others at Syracuse University are currently 
exploring the concept of relevance in the context of the sense-making 
approach. rS2, 23, 243 In their view, information need is how users see 
information helping them most effectively make sense of their individual 
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realities. Relevance is the user^s connection to these realities. The Milan 
and Eisenberg work is based on several assuMptions: 

(1) The user is an active creator of his own reality in a constantly 
changing environment; 

(2) Relevance is a mult idiaensional concept involving a relationship 
between a user and the user^s situation or environment; 

(3) Relevance is a cognitive phenomenon that is highly intersubject ive, 
but nevertheless systematic and measurable; and 

(4) The user is the ultimate judge of relevance; the dimensions of 
relevance can only be explored from the user^s perspective. 

This approach is wholly user-oriented or "user-centric." It looks at 

users entirely in the context of their own situations. Their situations 

include, among othr?r things, the information retrieval system. Because this 

approach builds exclusively cr^ environments and values as perceived by the 

user< an appropriate technique for exploring the concept of relevance is to 

allow users themselves to generate their own definitions of relevance, that 

is, of the "connections" between information and information need, ^ata on 

these connections can be collected through open-ended questioning, and 

analyzed through methods such as content analysis to uncover underlying 

patterns. C24, 253 

In 19B7, Nilan and Eisenberg C243 initiated an exploratory project with 
three different experimental settings: a focus group, individual interviews, 
and a supervised judgment task. In all three settings, users were asked to 
concentrate on their own real-life situations and to describe in their own 
words the connections they perceived between their information needs and the 
information retrieved. Responses generated in the focus group and individual 
interviews were tape-recorded and content-analyzed. The result was a list of 
39 "qualities of relevance," or descriptions of relationships between 
information needs and products. Finally, in the task setting, subjects were 
given this list and asked to choose the qualities that applied to their 
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individual situations, then to rank the qualities they chose in order of 
importance. 

Responses to the judgment task shoMed that the list Has so complete 
that the open-ended "other** category was scarcely necessary. Among the 
''connections** on the list were statements that the information or document 

was available, had a good index, was current/timely, or had 
authority/prestige. With regard to themselves and their situations, subjects 
stated that the information gave evidence or authority for a position, 
solved the problem, was interesting or worth knowing, was esthetically 
pleasing, reduced anxiety, stimulated their imagination, or that, 
serendipitously, **! stumbled on something interesting." 

Many of the factors Cooper and others had already suggested as 
contributing to relevance judgments, such as esthetics, appeared 
spontaneously in subjects' statements using this method. The factors that 
were gathered point to a definition of relevance both as a multidimensional 
and a systematic concept. The implication is that it is possible to 
establish a coherent set of evaluation criteria generated by users 
themselves. These criteria can then be incorporated in information retrieval 
system design. 

The user-centric study was, again, exploratory. Subject pools were 
small, for instance, and biased toward academic information needs. But while 
Milan and Eisenberg are still in the process of designing a large-scale 
project to test a user-centric definition of relevance, their early results 
show dramatic potential for the use of the user-centric model and 
appropriate qualitative analyses. 

CONCLUSION 

Until recently, little has been done to address the theoretical lack of 
connection between the realities of users and the realities of information 





systeMs. Inforsiation science still has no definition of relevance nor a 
conclusive understanding of the role of the user in relevance judgment. 

In this paper Me have, however, introduced an approach that can serve 
as a foundation for exploration. The assumptions underlying the user-centric 
vieif of relevance extend beyond viewing the concept as a multidimensional, 
active, systematic phenomenon. It suggests fundamental changes in the 
theoretical approaches of information science. The communication paradigm 
(the source-to-destination model of information retrieval) is too linear, 
too system-oriented, to provide a realistic understanding of the human 
judgment process. The user-centric approach suggests new definitions not 
just for relevance, but for its related concepts of information retrieval 
system, information need, and information environment. Perhaps most 
important is a reconceptualizat ion of information itself as something 
actively created by the individual. 
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